Method and apparatus for displaying light field video data

ABSTRACT

A method and an apparatus for displaying light field video data are described. A receiving unit of the apparatus receives light field video data including a plurality of light field video sequences. Each light field video frames of of the light field video sequences has multiple selectable viewpoints. A light field video frame is provided by an operation unit and displayed on a display device with a first viewpoint. An input unit receives an input indicating a viewpoint different from the first viewpoint. The displayed light field video frame is then displayed with the indicated viewpoint. In one embodiment, two light field video frames of two light field video sequences are displayed respectively in main and side windows on the display device.

TECHNICAL FIELD

The present invention relates to a method and an apparatus fordisplaying light field video data, particularly with differentviewpoints of the light field video data. The invention also relates toa computer readable medium suitable for such a method and an apparatus.

BACKGROUND

Light field capture data, which can be captured for example by aplenoptic camera or an array of cameras, allows refocusing of thecapture after the snapshot, estimating the depth of the scene in thecapture, and many other flexible editing options for the capture data.In order to visualize such light field content, several user interfaceshave been proposed to display and enable an operator to review and editindividual images of the light field capture [I].

Besides individual light field images, there are already devices able tocapture light field videos, for example, the Raytrix plenoptic camera.It can be expected that more light field videos will be available andused in the near future. However, the existing user interfaces are onlyapplicable to the visualization of light field images and are notdesigned to visualize light field videos. There is a need for a solutionthat enables users to review and make use of light field videos.

SUMMARY

Therefore, it is an objective to propose solutions for displaying andvisualizing light field video data. The proposed method and systemprovide solutions for displaying light field video data, as well as easyand intuitive navigation for reviewing the same.

According to one embodiment, a method for displaying light field videodata is proposed. The light field video data includes a plurality oflight field video sequences. Each light field video frame of the lightfield video sequences has multiple selectable viewpoints. The methodcomprises: receiving the light field video data including a plurality oflight field video sequences; displaying a first light field video frameof a first light field video sequence with a first viewpoint; receivingan input indicating a viewpoint of the first light field video framedifferent from the first viewpoint; and in response to the indicatinginput, displaying the first light field video frame with the indicatedviewpoint.

In one embodiment, the light field video frames of the first light fieldvideo sequence are temporally sequenced, and the method comprises: inresponse to the indicating input, displaying the light field videoframes of the first light field video sequence temporally subsequent tothe first light field video frame with the indicated viewpoint.

In one embodiment, the received input indicating the viewpoint isgenerated by selecting a region of the displayed first light field videoframe corresponding to the indicated viewpoint.

In one embodiment, the first light field video frame of the first lightfield video sequence is displayed in a main window on a screen. Themethod further comprises displaying a second light field video frame ofa second light field video sequence with a second viewpoint in a sidewindow on the screen.

In one embodiment, the first and second displayed light field videoframes are synchronized temporally or by other selectablecharacteristics derived from the contents of the light field video data.

In one embodiment, the method further comprises: receiving an inputselecting the displayed second light field video frame with the secondviewpoint in the side window; and, in response to the selecting input,(a) displaying the second light field video frame with the secondviewpoint in the main window, replacing the first light field videoframe, when the second viewpoint is in close proximity to the firstviewpoint, or (b) displaying the first light field video frame in themain window with a changing viewpoint, producing a smooth transitionover time between the first and second viewpoints, when the secondviewpoint is not in close proximity to the first viewpoint. Smoothingcan either be linear or done according to some user-defined curve,spline, etc. that will slow down or accelerate the transition from firstviewpoint to the second viewpoint over time.

According to one embodiment, an apparatus configured to provide lightfield video data to a display device is introduced. The apparatuscomprises a receiving unit, an operation unit, an input unit, andoptionally a storage unit. The receiving unit is configured to receivelight field video data including a plurality of light field videosequences. Each light field video frame of the light field videosequences has multiple selectable viewpoints. The storage unit isconfigured to store the received light field video data. The operationunit is configured to provide a first light field video frame of a firstlight field video sequence with a first viewpoint to a display device.The input unit is configured to receive an input indicating a viewpointof the first light field video frame different from the first viewpoint.The operation unit is configured to provide the first light field videoframe with the indicated viewpoint to the display device in response tothe indicating input.

In one embodiment, the light field video frames of the first light fieldvideo sequence are temporally sequenced, and the operation unit isconfigured to provide the light field video frames of the first lightfield video sequence temporally subsequent to the first light fieldvideo frame to the display device with the indicated viewpoint inresponse to the indicating input.

In one embodiment, the operation unit is configured to provide a secondlight field video frame of a second light field video sequence with asecond viewpoint to the display device. Preferably, the operation unitis configured to temporally synchronize the first and second light fieldvideo frames.

In one embodiment, the input unit is configured to receive an inputselecting the second light field video frame with the second viewpoint.The operation unit is configured to, in response to the selecting input,provide to the display device (a) the second light field video framewith the second viewpoint when the second viewpoint is in closeproximity to the first viewpoint, or (b) the first light field videoframe with a changing viewpoint, producing a smooth transition betweenthe first and second viewpoints, when the second viewpoint is not inclose proximity to the first viewpoint.

Also, a computer readable storage medium has stored therein instructionsenabling displaying light field video data, which, when executed by acomputer, cause the computer to: receive light field video dataincluding a plurality of light field video sequences; display a firstlight field video frame of a first light field video sequence with afirst viewpoint; receive an input indicating a viewpoint of the firstlight field video frame different from the first viewpoint; and inresponse to the indicating input, display the first light field videoframe with the indicated viewpoint.

The computer readable storage medium can be a non-transitory volatile ornon-volatile storage medium, such as, for example, a hard disk, anoptical or magnetic disk or tape, a solid state memory device, etc. Thestorage medium thus tangibly embodies a program of instructionsexecutable by a computer or a processing device to perform program stepsas described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding the proposed solutions shall now be explainedin more detail in the following description with reference to thefigures. It is understood that the solutions are not limited to thisdisclosed exemplary embodiments and that specified features can alsoexpediently be combined and/or modified without departing from the scopeof the proposed solutions as defined in the appended claims.

FIG. 1 is a schematic diagram illustrating an exemplary embodiment of anapparatus configured to provide light field video data to a displaydevice.

FIG. 2 is a flow chart illustrating a preferred embodiment of a methodfor displaying light field video data.

FIG. 3 schematically illustrates an exemplary display of light fieldvideo data on a display device according to an embodiment of the method.

FIG. 4 schematically illustrates a viewpoint trajectory of the displayof LF video data according to one embodiment of the method.

FIG. 5 displayed on a display device according to one embodiment of themethod.

FIG. 6 is a flow chart illustrating another preferred embodiment of amethod for displaying light field video data.

FIG. 7 schematically illustrates the display transition from one lightfield video data to another light field video data according to oneembodiment of the method.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The term “light field video data” (hereinafter LF video data) used inthe description should be understood as generally defined in the field,i.e. 4D video data comprising, at any instant, angular and spatialinformation about all the light rays at every point travelling in everydirection in a space scene, and can be captured by, for example, aplenoptic video camera, a video camera array, or any other feasiblecamera device.

FIG. 1 schematically illustrates an exemplary embodiment of an apparatus10 configured to provide light field video data to a display device 15.The apparatus 10 comprises a receiving unit 11, an operation unit 12, aninput unit 13 and optionally a storage unit 14.

The receiving unit 11 is configured to receive LF video data including aplurality of LF video sequences. Each frame of the LF video sequenceshas multiple selectable viewpoints. The LF video data can be capturedand provided by a plenoptic camera to the apparatus 10. In alternative,the receiving unit 11 can retrieve and receive the LF video data from adatabase (not shown).

The received LF video data can be optionally stored in a storage unit14, which can be any feasible and suitable memory device and can beeither integrated in or supplementary to the apparatus 10.

The operation unit 12 is configured to retrieve light field video framesfrom the receiving unit 11 or the storage unit 14 and to provide thesame to a display device 15. The display device 15 can be a display unitintegrated in the apparatus 10 or an external display device outside theapparatus 10. The display device 15 can be, for example, a television, acomputer screen, a mobile device with display capabilities, etc.

The input unit 13 is configured to receive an input from a user or anoperator choosing and indicating viewpoints of the LF video frames. Auser can indicate a specific viewpoint of a certain LF video frame viathe input unit 13. In response to and according to the inputinstruction, the operation unit 12 is configured to retrieve the chosenLF video frame with the indicated viewpoint and to provide the same tothe display device 15. The input unit 13 provides an interface enablinga user to interact with and control the apparatus 10. The types and thedesign of the input unit 13 can be flexible and varied for differentdemands and purposes.

Preferably, the operation unit 12 is further configured to providemultiple LF video frames temporally subsequent to the chosen LF videoframe to the display device 15 in response to the indicating input, allof which being with the indicated viewpoint.

Optionally, the input unit 13 can be combined with the display device15. For example, the display device 15 can be a touch screen thatdisplays the LF video frames and at the same time enables a user toenter an input. In this case, a user can indicate the viewpoint of avideo frame directly using the touch screen 15, rather than using asupplementary input unit 13.

In one embodiment, the operation unit 12 is configured to provide afirst LF video frame of a LF field video sequence with a first viewpointand a second LF video frame of a second LF video sequence with a secondviewpoint to the display device 15. Preferably, the first and second LFvideo frames are temporally synchronized by the operation unit 12.

More details about the operation of the apparatus 10 will be describedbelow together with the proposed methods for displaying LF video data.Of course, the apparatus 10 can optionally comprise any other additionalor alternative devices for the implementation of the embodiments of thefollowing proposed methods.

FIG. 2 schematically illustrates a preferred embodiment of the methodfor displaying LF video data. The LF video data discussed hereinpreferably includes a plurality of LF video sequences, and each LF videoframe of the LF video sequences has multiple selectable viewpoints. Themethod comprises: receiving 20 the LF video data; displaying 21 a firstLF video frame of a first LF video sequence with a first viewpoint;receiving 22 an input indicating a viewpoint of the first LF video framedifferent from the first viewpoint; and, in response to the indicatinginput, displaying 21 the LF video frame with the indicated viewpoint.

Referring to FIG. 1, one preferred embodiment of the method will befurther explained in detail below using the exemplary embodiment of theapparatus 10. It should be understood that the method can be implementedin any other suitable systems or apparatuses known and available in thefield.

The receiving unit 11 of the apparatus 10 receives 20 the LF video dataincludes one or more LF video sequences from a camera or a LF videodatabase. Each of the LF video sequences includes a plurality of LFvideo frames having multiple selectable viewpoints. For example, for anevent such as a football game or a wedding, several LF video sequencesrelating to a same event will be available from different sources orcameras. Preferably, these LF video sequences are related to each otherin certain content, and there are some semantical similarities amongthem.

The received LF video data can be optionally stored in the storing unit14 and be used afterwards. The operation unit 12 retrieves the LF videodata and the LF video sequences and frames thereof and provides the sameto the display device 15, e.g., a screen, to be displayed 21.

Preferably, the default display of the LF video data shows multiple LFvideo sequences respectively. In particular, each of the LF videosequence is displayed 21 in a certain window or area on a screen 15.FIG. 3(a) schematically illustrates one exemplary default display ofmultiple LF video sequences on a screen 15. Sixteen LF video data A, B,C, etc., are shown on the screen 15, where each one is displayed 21 in arespective rectangular window. The LF video sequences can be showndynamically with a continuous video stream or be represented by one LFvideo frame of the corresponding LF video sequence.

Preferably, the LF video sequences are displayed 21 all-in-focus in thedefault display. Of course, the amount of the LF video sequences shownin the default display is flexible and can be changed and arbitrarilychosen by a user for different demands and preferences.

In order to display one specific LF video sequence shown in the defaultdisplay, a user can enter an input indicating a chosen LF video sequencevia the input unit 13 of the apparatus 10 or directly via the screen 15if it is a touch screen. The selection of one specific LF video sequencecan be easily performed by, for example, using a mouse and acorresponding cursor on the screen 15 to click on the window of thetarget LF video sequence. When the screen 15 is a touch screen, the usercan touch the window directly with his/her fingers. Any other existingand general remote control and/or user input devices can be used aswell.

Referring to FIG. 3(b), the display of the LF video data “A” on thescreen 15 is schematically illustrated, in response to an inputindicating a selection of the LF video sequence A. A first LF videoframe of the LF video sequence A with a first viewpoint is displayed 21as a default setting, which is preferably the viewpoint closest to theaverage viewpoint of the corresponding LF video sequence. For example,the LF video sequence A can be displayed with the viewpoint A₁ as adefault. Without a further input and/or instruction, it is preferredthat the LF video sequence A is displayed frame by frame with thedefault viewpoint A₁. In other words, the LF video data A is displayedas a video stream on the screen 15 with the viewpoint A₁.

The multiple selectable viewpoints for a LF video sequence and theframes thereof are one of the advantages and improvements of LF videodata. Accordingly, in the preferred embodiment of the proposed methodfor displaying LF video data, the viewpoint of a displayed LF videosequence is especially selectable and changeable. For example, as shownin FIG. 3, several selectable viewpoints associated with the videosequence A are shown on the screen 15. In this example, sixteenviewpoints A₁, A₂, A₃ . . . are available for a user to select from. Itshould be understood that the amount of the available viewpoints is notlimited and can be flexibly decided.

A user can enter an input indicating and selecting one specificviewpoint of the displayed first LF video frame of the LF video sequenceA. For example, the input can indicate the viewpoint A₇, which isdifferent from the default viewpoint Al. In response to the reception 22of the indicating input, the operation unit 12 provides the first LFvideo frame with the indicated viewpoint A₇ to the display device 15.Then the first LF video frame of the LF video sequence A is displayed 21with the indicated viewpoint A₇ on the screen 15.

FIG. 4 shows a viewpoint trajectory of the display of the exemplary LFvideo sequence A. At timing T₁ the first LF video frame of the LF videosequence A is displayed with the viewpoint A₇ according to the reception22 of the indicating input as described above. At a following timing T₂when a subsequent video frame is displayed, a user enters an input toindicate and select another viewpoint A₁₃, resulting in the display ofthis LF video frame with the indicated viewpoint A₁₃. If no furtherinput is received, as shown in FIG. 4, the LF video sequence A iscontinuously displayed frame by frame with the last viewpoint A₁₃. Inother words, the LF video frames of the LF video sequence A aredisplayed in a temporal order with the indicated viewpoint.

According to the preferred embodiment of the proposed method, a user canchange the viewpoint of the display of the LF video sequences at anytime. It should be noted that the division of the screen 15 to show theselectable viewpoints can be either visible or invisible for the user.The dotted lines and areas marked in grey in FIGS. 3 and 4 are used heremerely for a clear explanation in the description and do not need to bevisible on the screen 15. A user can also select the viewpoints withoutthe visualization of the viewpoints simply by clicking on and indicatinga specific area or region of the displayed LF video frame correspondingto a specific viewpoint. For example, a user can be educated previouslyabout the setting and visualization of the LF video sequences on thescreen 15, in which case the user would know how to enter an input toindicate a preferred viewpoint.

Alternatively, when the display device 15 is a touch screen, the usercan select a new viewpoint using a swiping finger gesture on the inputunit 13. The processing unit 12 then determines the new viewpoint basedon the direction and amplitude of the swipe gesture. Specifically, thenew viewpoint is chosen in such a way that the direction of theviewpoint change between the initial viewpoint and the newly selectedviewpoint matches the direction of the swipe gesture, and that theamplitude of the viewpoint change is proportional to the amplitude ofthe swipe gesture.

In addition to the selection and change of the viewpoints of thedisplayed LF video sequences, the focus of the same can also be chosenand changed. In one embodiment, the LF video sequences are displayedall-in-focus on the screen 15, preferably as a default display of the LFvideo sequences. In another embodiment, a user can decide which objectto be focused in one LF video sequences and/or where is the focus planein the scene in the LF video sequences by indication operations similarto those described above. Specifically, the user can select one area orone object in the displayed LF video sequences by the input unit 13 ordirectly touching a touch screen 15 when it is feasible. This refocusingoperation can be triggered by a selection of the “Re-focus” item shownon the screen 15. A gyroscope can also be used in order to change thefocus of the displayed LF video data.

In one embodiment, a user can also pause the display of the LF videodata. For example, referring to FIG. 3, a user can trigger the pauseoperation by clicking on the “pause” item shown on the screen 15. Withthe possibility to pause the display of the LF video sequences, itbecomes easier for a user to change the viewpoints or the focus duringthe pause, which is especially interesting regarding a sport LF video ora LF video with changing content. The user can stop the display of a LFvideo sequence when an action of interest is performed, e.g., a goalattempt in a football game, and change the viewpoint and/or the focus ofthe displayed frame. In this way, the user can further enjoy the videoLF data with a preferred viewpoint and focused region, which improvesthe viewing experience. Of course, the pause operation can be performeddifferently depending on different apparatus 10, the input unit 13thereof, and the display device 15. For example, when the display device15 is a touch screen, the user could pause the display by simply doubletouching the screen 15.

In one embodiment, a function of stabilization can also be activatedand/or triggered by a user, for example, by selecting the “Stabilize”item displayed on the screen 15 as shown in FIG. 3. In this way, angularinformation can be used to remove a shaky movement of a LF videosequence captured by a hand-held camera, which can be performed, forexample, by using available methods in the field [II].

In one embodiment, the user can focus his/her attention on a particularobject of interest in the scene captured by the LF camera. In this case,it is preferred to provide an operation for drawing a bounding boxaround a target object of interest in some reference LF views,preferably near the central viewpoint, and tracking the bounding boxthroughout several views, based on LF calibration data and visualtracking methods available and known in the field [III]. Relying on theobject tracking information, the operation unit 12 of the apparatus 10can generate from one target LF video sequence a monocular video wherethe object of interest is located as close as possible to the center ofthe displayed LF video sequence or to its initial position in thereference view. This can be achieved by dynamically selecting theappropriate viewpoint, among those made available by the light LF videodata.

Specifically, the bounding box drawn by the user in at least onereference view provides a template image patch. This same image patch isalso present in the other views of the LF data, but at displacedlocations with respect to the reference view, because of the change ofthe viewpoint. The displacement of the image patch between the referenceview and any other view available in the light field sequence can beestimated by methods known from the state of art and described in [III],such as normalized cross-correlation. Considering a given second viewdifferent from the reference view, adding as a vector the computeddisplacement for the second view to the location of the patch center inthe reference view yields an estimate of the location of the templatepatch in the second view. In the proposed embodiment, the new viewpointis computed by selecting, among all available views in the LF sequence,the view for which the computed location of the center of the templateimage patch is closest to the center of the image.

Furthermore, this bounding box, once tracked, may be used to automatethe modification of the viewpoint in successive LF video frames. Inother words, the new viewpoints in successive frames will be the ones onwhich the best focus can be found within the tracked bounding box. Thisway, the viewpoints will always be focused on the tracked object ofinterest.

In another preferred embodiment of the proposed method, the first LFvideo frame of the first LF video sequence is displayed in a main windowon the screen 15. The method further comprises displaying 21 a secondlight field video frame of a second light field video sequence with asecond viewpoint in a side window on the screen.

FIG. 5 schematically illustrates the described main and side windowsdisplayed on an exemplary screen 15 in a preferred embodiment of theproposed method. In this exemplary illustration, one main window andseven side windows are shown, which are in the shape of rectangles. Ofcourse, the size, shape, and the amount of the main and side windows areflexible and not limited. Accordingly, the amount of the LF video datashown in the side windows is also not limited and can be flexiblydecided. Preferably, the size of the main window is larger than the sizeof the side windows, and the main window is positioned closer to thecenter of the screen than the side windows. In this way, the main windowcan be more easily seen by a user and attracts more attention than theside windows.

Preferably, the displayed first and second LF video frames of the firstand second LF video sequences have similar content or at least somesimilar objects in the video sequences. In addition, the relativeposition of the main window and the side windows on the screen 15 ispreferably related to the content of the first and second LF videoframes displayed therein, respectively. For example, a second LF videoframe X is displayed and placed at the right-hand side of the mainwindow when there is a right displacement between the viewpoint of thefirst LF video frame displayed in the main window and the viewpoint ofthe second LF video frame X displayed in the side window. The sidewindows placed at the top, bottom or left-hand side of the main windoware similarly arranged.

In alternative, the main and side windows can also be arranged accordingto other similarities between the displayed first and second LF videoframes, such as colorimetry of the frames, global shapes similarities,time similarities, detected moves similarities, environmentsimilarities, presence of same people in the frames as evidenced by facedetection and face recognition, specified or detected objectssimilarities, etc. In addition, the similarities can also be defined anddecided by the operation unit 12 of the apparatus 10 according to anysuitable and feasible parameters and/or metadata acquired together withthe LF video data.

In one embodiment, the first and second displayed LF video frames arepreferably temporally synchronized, i.e. the displayed first and secondLF video frames have a same timecode and/or have been captured at a sametime. The synchronization of the displayed LF video frames can also beperformed according to other characteristics of the video frames, suchas capturing locations, people in the frames, audio, etc [IV]. Thesynchronization can be easily activated or deactivated by a userselecting the “synchronize” item on the screen 15 using the input unit13 of the apparatus 10 or the display device 15 directly, similar to theactivation/deactivation of other functions as described above.

Moreover, the first and second LF video frames can be optionallydisplayed all-in-focus, preferably as a default display. The focus ofthe displayed LF video frames can be arbitrarily changed and selected asdescribed above with the use of the “Re-focus” item shown on the screen15.

Referring to FIG. 6, in one preferred embodiment, the method furthercomprises: receiving 22 an input indicating a viewpoint of the displayedfirst LF video frame in the main window; receiving 23 an input selectingthe displayed second LF video frame with the second viewpoint in theside window; and in response to the selecting input, (a) displaying 21the second LF video frame with the second viewpoint in the main window,replacing the first LF video frame, when the second viewpoint is inclose proximity to the first viewpoint, or (b) displaying 21 the firstLF video frame in the main window with a changing viewpoint when thesecond viewpoint is not in close proximity to the first viewpoint. Thetransition from one LF video data to another LF video data is denoted as“Video Jump” hereinafter, of which the transition path is schematicallyillustrated in FIG. 7 in detail.

FIG. 7(a) illustrates the first situation, where the second viewpoint ofthe second LF video frame (right-hand side) displayed in the side windowis in close proximity to the first viewpoint of the first LF video frame(left-hand side) displayed in the main window. At the time instant T₁,the first LF video sequence is displayed in the main window on thescreen with a specific viewpoint (marked in grey in the figure). Attiming T₂, the displayed viewpoint of the first LF video sequence ischanged by using, for example, the method described above. The changedviewpoint is then in close proximity to a viewpoint of the second LFvideo frame of the second LF video sequence, which is displayed in aside window next to the main window on the screen 15. In this case, inresponse to a user input selecting the second LF video frame, thecorresponding second LF video sequence can be displayed in the mainwindow, replacing the first LF video sequence. Accordingly, at timingT₃, the second LF video sequence is displayed in the main window(possibly with a same second viewpoint or a further changed viewpoint asshown in the figure), and the first video sequence is preferablydisplayed in a side window instead.

The selection of the second LF video sequence can be performed, forexample, via the exemplary input unit 13 of the apparatus 10. In thecase where the display device 15 is a touch screen, a user can simplyclick on the side window displaying the second LF video sequence andsweeps it to the main window. It is preferred that the Video Jumpoperation is seamless, which means that the display of the first LFvideo sequence in the main window is not paused during the transition ofthe display from the first LF video sequence to the second LF videosequence. A gyroscope can also be used for such a seamless Video Jumpoperation.

FIG. 7(b) shows another situation, where the second viewpoint of theselected second LF video frame of the second LF video sequence is not inclose proximity to the first viewpoint of the displayed first LF videoframe of the first LF video sequence at the time of the selection of theVideo Jump operation. In this case, it is still preferred that thetransition of the LF video sequences is performed smoothly in terms ofviewpoints. That is, the first LF video sequence is displayed in themain window with a changing viewpoint until the viewpoint of the firstLF video sequence is in close proximity to the second viewpoint of thesecond LF video sequence. As shown in FIG. 7(b), the Video Jumpoperation is triggered at T, resulting in a changing viewpoint of thefirst LF video data throughout the timings T+1, T+2 and T+3. At thetiming of T+3, the viewpoint of the first LF video sequence becomes inclose proximity to the second viewpoint of the displayed second LF videosequence, which leads to the display of the second LF video sequence inthe main window as described in the previous situation.

In one embodiment, the Video Jump operation can also be performed withthe pause function as described above. For example, in the case of agoal attempt in a soccer game, a user might want to watch other LF videodata from other viewpoints capturing the same goal post. In this case,the Video Jump is also possible to be operated within the pause mode. Inother words, the Video Jump operation can also be performed with thepause operation. The transition of the LF video sequences is stillpreferably performed smoothly in terms of viewpoints. When the LF videois played again, the visualization starts from the last LF videosequence watched.

In one embodiment, the proposed method for displaying LF video data canbe used integrated with available broadcasting techniques and methods.For example, when a user can get access to a set of broadcast LF videosthat are related to a same live event, e.g. a soccer match, a Formula 1Race, a stage of the Tour de France, etc., the proposed method can beapplied and implemented provided that the display apparatus embedsmultiple tuners and time-shifting features. The display and navigationamong the several broadcast LF video signals is possible, as well as the“pause” and other operations mentioned above. One exemplary case is thatwhen the user browses the LF videos or manipulates them during a “pause”operation, the broadcast live LF video signals can be recorded andplayed back or resumed at a later time.

In one embodiment, the proposed method for displaying LF video data canbe used by a content provider to edit broadcast or broadband content. Onthe back-end side of a professional services provider, many LF videodata can be used as inputs to edit final LF video data. These finalvideo data can then either be stored for later broadcast/broadband orcan be broadcast on the fly, with maybe time shifting. Furthermore, oncefinal LF video data have been edited the end-user, professional as wellas consumer one, can either receive the final LF video data on abroadcast channel or request for its playing on demand via a broadbandchannel.

REFERENCE

-   -   [I] LightField Forum http://nb.dropmark.com/89973/1342251    -   [II] B. M. Smith, et al., “Light Field Video Stabilization”,        IEEE International Conference on Computer Vision (ICCV), 2009    -   [III] A. W. M. Smeulders, et, al. “Visual Tracking: An        Experimental Survey”, IEEE Transactions on Pattern Analysis and        Machine Intelligence, Volume: 6, Issue: 7, 2014    -   [IV] A. Bagri, et al., “A Scalable framework for joint        clustering and synchronizing multi-camera videos”, 21^(st)        European Signal Processing Conference (EUSIPCO 2013), September        2013

1. A method for displaying light field video data, comprising: receivinglight field video data including a plurality of light field videosequences, each light field video frame of the light field videosequences having multiple selectable viewpoints; displaying a firstlight field video frame of a first light field video sequence with afirst viewpoint; receiving an input indicating a viewpoint of the firstlight field video frame different from the first viewpoint; and inresponse to the indicating input, displaying (21) the first light fieldvideo frame with the indicated viewpoint.
 2. The method of claim 1,wherein the light field video frames of the first light field videosequence are temporally sequenced, and wherein the method comprises: inresponse to the indicating input, displaying the light field videoframes of the first light field video sequence temporally subsequent tothe first light field video frame with the indicated viewpoint.
 3. Themethod of claim 1, wherein the received input indicating the viewpointis generated by selecting a region of the displayed first light fieldvideo frame corresponding to the indicated viewpoint and/or using aswipe gesture indicating a direction and an amplitude corresponding tothe indicated viewpoint.
 4. The method of claim 2, wherein the receivedinput indicating the viewpoint is generated by selecting a region of thedisplayed first light field video frame corresponding to the indicatedviewpoint and/or using a swipe gesture indicating a direction and anamplitude corresponding to the indicated viewpoint.
 5. The method ofclaim 1, wherein the first light field video frame of the first lightfield video sequence is displayed in a main window on a screen, themethod further comprising: displaying at least one second light fieldvideo frame of a second light field video sequence with a secondviewpoint in at least one side window on the screen.
 6. The method ofclaim 5, wherein the relative position of the main and side windows onthe screen is related to the content of the first and second displayedlight field video frames.
 7. The method of claim 5, wherein the firstand second displayed light field video frames are temporallysynchronized.
 8. The method of claim 5, wherein the size of the mainwindow is larger than the size of the side window and/or the main windowis positioned closer to the center of the screen than the side window.9. The method of claim 5, further comprising: receiving an inputselecting the displayed second light field video frame with the secondviewpoint in the side window; and in response to the selecting input,displaying the second light field video frame with the second viewpointin the main window replacing the first light field video frame when thesecond viewpoint is in close proximity to the first viewpoint, ordisplaying the first light field video frame in the main window with achanging viewpoint when the second viewpoint is not in close proximityto the first viewpoint.
 10. The method of claim 1, wherein the firstlight field video frame is displayed all-in-focus.
 11. The method ofclaim 5, wherein the first and/or the second light field video framesare displayed all-in-focus.
 12. The method of claim 1, wherein the focusplane of the first light field video frame of corresponding light fieldvideo sequence is selectable.
 13. The method of claim 5, wherein thefocus plane of the first and/or the second light field video frames ofcorresponding light field video sequences is selectable.
 14. Anapparatus configured to provide light field video data to a displaydevice, comprising: a receiving unit configured to receive (20) lightfield video data including a plurality of light field video sequences,each light field video frame of the light field video sequences havingmultiple selectable viewpoints; an operation unit configured to providea first light field video frame of a first light field video sequencewith a first viewpoint to a display device; and an input unit configuredto receive (22) an input indicating a viewpoint of the first light fieldvideo frame different from the first viewpoint, wherein, in response tothe indicating input, the operation unit is configured to provide thefirst light field video frame with the indicated viewpoint to thedisplay device.
 15. The apparatus of claim 14, further comprising astorage unit configured to store the received light field video data.16. The apparatus of claim 14, wherein the light field video frames ofthe first light field video sequence are temporally sequenced, and, inresponse to the indicating input, the operation unit is configured toprovide to the display device the light field video frames of the firstlight field video sequence temporally subsequent to the first lightfield video frame with the indicated viewpoint.
 17. The apparatus ofclaim 14, wherein the operation unit is configured to provide a secondlight field video frame of a second light field video sequence with asecond viewpoint to the display device and to temporally synchronize thefirst and second light field video frames.
 18. The apparatus of claim17, wherein the input unit is configured to receive an input selectingthe second light field video frame with the second viewpoint, andwherein in response to the selecting input, the operation unit isconfigured to provide to the display device the second light field videoframe with the second viewpoint when the second viewpoint is in closeproximity to the first viewpoint, or the first light field video framewith a changing viewpoint when the second viewpoint is not in closeproximity to the first viewpoint.
 19. A computer readable storage mediumhaving stored therein instructions enabling displaying light field videodata, which, when executed by a computer, cause the computer to: receivelight field video data including a plurality of light field videosequences, each light field video frame of the light field videosequences having multiple selectable viewpoints; display a first lightfield video frame of a first light field video sequence with a firstviewpoint; receive an input indicating a viewpoint of the first lightfield video frame different from the first viewpoint; and in response tothe indicating input, display the first light field video frame with theindicated viewpoint.